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Abstract 


The  Helix  project  is  a  multi-year  investigation  of  the  demographics  and  effectiveness  of  systems  engineers  in  the 
US  Department  of  Defense  (DoD)  and  the  defense  community.  This  project  is  sponsored  by  the  DoD  and  the 
National  Defense  Industrial  Association  -  Systems  Engineering  Division  (NDIA-SED),  and  has  been  initiated  in 
response  to  the  many  challenges  concerning  the  systems  engineering  (SE)  workforce.  This  report  highlights  the 
project  activities  since  September  2012  until  April  2013  in  preparation  for  further  data  collection  and  analysis  to 
be  performed  from  May  2013  onwards. 

Initial  months  of  the  Helix  project  have  been  spent  in  formulating  the  research  approach;  deciding  the  data 
collection  methodology;  preparing  the  instruments  for  data  collection;  obtaining  necessary  clearances  and 
permissions;  setting  up  and  interacting  with  the  advisory  panel;  presenting  the  Helix  project  to  various  systems 
engineering  groups  and  professional  societies  to  identify  participating  organizations;  interacting  with  points-of- 
contacts  of  individual  organizations  towards  further  data  collection;  setting  up  protocols  and  procedures  to 
handle  data  in  a  secure  manner;  obtaining  IRB  and  DoD  approvals;  and  selecting  software  tools  for  qualitative  and 
quantitative  data  analysis. 

From  May  2013  onwards,  the  project  team  expects  to  begin  conducting  actual  data  collection,  perform  extensive 
qualitative  and  quantitative  data  analysis,  and  to  then  report  findings  periodically. 


Contract  Number:  H98230-08-D-0171  Page  3  WHS  TO  028  RT  45 

Report  No.  SERC-2013-TR- 038-1 
May  20,  2013 

UNCLASSIFIED 


Table  of  Contents 


Abstract . 3 

Table  of  Contents . 4 

Introduction . 5 

The  Challenge . 5 

The  Response . 5 

Governance  and  Project  Team . 5 

Project  Activities  and  Status  [Sep  2012  -  Apr  2013] . 6 

Research  Approach  and  Data  Collection . 6 

Helix  Advisory  Panel  (HAP) . 6 

Gathering  Participating  Organizations . 7 

Permissions,  Protocols,  and  Data  Handling . 7 

Project  Plans  [May  2013  onwards] . 8 

Appendix  A:  Sample  Interview  Questions  (Individual  Level) . 9 

Appendix  B:  Sample  Interview  Questions  (Organizational  Level) . 10 

Appendix  C:  Institutional  Data . 11 


Contract  Number:  H98230-08-D-0171  Page  4  WHS  TO  028  RT  45 

Report  No.  SERC-2013-TR- 038-1 
May  20,  2013 

UNCLASSIFIED 


Introduction 


The  Helix  project  is  a  multi-year  investigation  of  the  demographics  and  effectiveness  of  systems  engineers  in  the 
US  Department  of  Defense  (DoD)  and  the  defense  community.  This  project  is  sponsored  by  the  DoD  and  the 
National  Defense  Industrial  Association  Systems  Engineering  Division  (NDIA-SED),  has  been  initiated  in  response 
to  the  many  challenges  concerning  the  systems  engineering  (SE)  workforce.  This  report  highlights  the  project 
activities  since  September  2012  until  April  2013  in  preparation  for  further  data  collection  and  analysis  to  be 
performed  from  May  2013  onwards. 


The  Challenge 

Many  systems  engineers  in  the  defense  workforce  are  nearing  retirement.  Budget  and  schedules  are  being 
dramatically  compressed.  Mission  requirements  demand  ever  more  sophisticated  and  complex  systems.  The 
tools,  processes,  and  technologies  that  systems  engineers  must  master  keep  changing  ever  more  rapidly. 
Organizations  respond  in  many  different  ways  to  these  winds  buffeting  their  systems  engineers.  Yet,  there  is  scant 
agreement  or  understanding  on  what  the  most  important  levers  are  that  affect  the  effectiveness  of  systems 
engineers,  how  to  measure  that  effectiveness,  and  the  impact  various  workforce  improvement  efforts  are  having. 


The  Response 

The  Helix  project  will  address  this  challenge  by  addressing  three  primary  research  questions: 

1.  What  are  the  characteristics  of  systems  engineers? 

2.  How  effective  are  systems  engineers  and  why? 

3.  What  are  employers  doing  to  improve  their  effectiveness? 

Answers  to  these  questions  will  be  obtained  by  analyzing  data  collected  in  two  primary  modes  for  the  project:  (1) 
face-to-face  interviews,  and  (2)  institutional  data  in  the  form  of  artifacts.  Analyses  of  the  data  will  be  reported  in 
an  aggregated  manner  without  revealing  the  identity  of  the  participants  or  their  organizations. 


Governance  and  Project  Team 

The  Helix  project  is  being  run  as  a  project  of  the  Systems  Engineering  Research  Center  (SERC),  a  DoD  University 
Affiliated  Research  Center.  To  ensure  Helix  delivers  the  greatest  value  and  to  help  Helix  obtain  access  to  the 
necessary  data,  DoD  and  NDIA-SED  have  formed  the  Helix  Advisory  Panel  (HAP)  with  representatives  from  both 
organizations  and  chaired  by  Dr.  Don  Gelosh,  Worcester  Polytechnic  Institute,  and  co-chaired  by  Mr.  Nicholas 
Torelli,  Office  of  the  Deputy  Assistant  Secretary  of  Defense  for  Systems  Engineering.  The  research  team  at  the 
Stevens  Institute  of  Technology  is  led  by  Dr.  Art  Pyster,  co-led  by  Dr.  Stan  Rifkin,  and  supported  by  Deva  Henry, 
Nicole  Hutchison,  and  Dr.  Kahina  Lasfer. 
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Project  Activities  and  Status  [Sep  2012  -  Apr  2013] 


Initial  months  of  the  Helix  project  have  been  spent  in  formulating  the  research  approach;  deciding  the  data 
collection  methodology;  preparing  the  instruments  for  data  collection;  obtaining  necessary  clearances  and 
permissions;  setting  up  and  interacting  with  the  advisory  panel;  presenting  the  Helix  project  to  various  systems 
engineering  groups  and  societies  to  gather  participating  organizations;  interacting  with  points-of-contacts  of 
individual  organizations  towards  further  data  collection;  setting  up  protocols  and  procedures  to  handle  data  in  a 
secure  manner;  obtaining  IRB  and  DoD  approvals;  and  selecting  software  tools  for  qualitative  and  quantitative 
data  analysis. 


Research  Approach  and  Data  Collection 

The  primary  mode  of  data  collection  would  be  face-to-face  interviews  with  systems  engineers  representing  a  wide 
cross-section  of  the  SE  workforce.  Two  broad  categories  of  interviewees  are  being  addressed  initially:  individual 
systems  engineers  who  can  provide  their  perspectives  on  the  research  question,  and  organizational 
spokespersons  who  can  provide  the  organization's  perspective  on  the  research  questions.  Resumes  /  career-briefs 
will  be  collected  from  all  interviewees,  with  proper  consent.  In  addition,  institutional  data  will  be  collected  from 
participating  organizations  to  better  understand  the  characterization,  structure,  organization,  and  improvement 
initiatives  of  the  SE  workforce.  Further  details  are  included  in  the  following  appendices: 

Appendix  A:  Sample  Interview  Questions  (Individual  Level) 

Appendix  B:  Sample  Interview  Questions  (Organizational  Level) 

Appendix  C:  Institutional  Data 

Data  collection  will  be  performed  in  three  rounds: 

Round  1:  Introduction  -  Interactions  with  organizational  points  of  contacts  to  clarify  research  objectives 
and  mutual  expectations,  and  to  establish  ground  rules.  Collection  of  institutional  data  will  be  initiated. 
Round  2:  Small  scale  interviews  -  Interviews  will  be  conducted  with  a  small  number  of  employees  (up  to 
6);  more  institutional  data  will  be  collected;  preliminary  data  analysis  will  be  performed  and  shared  with 
participating  organization 

Round  3:  Full  scale  interviews  -  Based  on  experiences  and  preliminary  data  analysis  from  Round  2, 
interviews  will  now  be  conducted  with  a  larger  number  of  individuals  from  the  organization  (about  50). 

Round  1  is  currently  in  progress  with  a  number  of  different  organizations  from  within  DoD  and  the  NDIA-SED 
companies.  It  is  expected  that  Round  2  will  be  performed  during  May-July  2013  and  Round  3  from  August  2013 
onwards.  It  is  expected  that  new  organizations  that  enter  the  Helix  project  will  still  go  through  Rounds  1-3. 


Helix  Advisory  Panel  (HAP) 

The  Helix  Advisory  Panel  (HAP)  has  been  formed  with  representatives  from  DoD  and  NDIA-SED;  and  chaired  by  Dr. 
Don  Gelosh  and  co-chaired  by  Mr.  Nicholas  Torelli.  The  HAP  will  be  responsible  for  providing  (a)  strategic  advice 
on  the  direction  of  Helix,  (b)  feedback  on  methodology  and  findings,  and  (c)  access  to  data  sources,  including 
people  and  databases. 
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Three  meeting  of  the  HAP  have  been  held  in  January,  February,  and  April  2013.  During  these  meetings,  the  project 
team  briefed  panel  on  progress  with  the  research  approach,  research  instruments,  and  status  of  interactions  with 
participating  organizations.  The  HAP  has  been  instrumental  in  providing  feedback  to  the  research  team  and  also  in 
identifying  venues  for  the  Helix  project  to  be  presented  to  various  SE  related  groups  and  forums  with  an  objective 
to  attract  more  participating  organizations. 

The  HAP  will  convene  next  in  June  or  July  2013  following  some  initial  interviews  and  data  analysis. 


Gathering  Participating  Organizations 

Presentations  about  the  Helix  project  have  been  delivered  at  meetings  of  various  groups  and  forums  including 
NDIA  Systems  Engineering  Division  (SED),  NDIA  SED  Education  and  Training  (E&T)  sub-committee,  INCOSE 
Corporate  Advisory  Board,  AIAA  Systems  Engineering  Technical  Committee  (SETC),  Navy  Systems  Engineering 
Stakeholders  Group  (SESG)  -  Systems  Engineering  Educational  Continuum  (SEEC)  Working  Group,  Army  RDECOM, 
and  Army  SE  Forum. 

The  Helix  project  expects  to  provide  the  following  value  to  all  participating  organizations: 

1.  Deepen  the  understanding  of  the  organization's  systems  engineering  workforce  over  time 

2.  Identify  gaps  and  know  where  to  focus  investments  to  improve  the  systems  engineering  workforce 

3.  Benchmark  the  organization's  systems  engineering  workforce 

4.  Deepen  the  understanding  of  what  makes  the  systems  engineering  workforce  effective 

5.  Receive  recommendations  on  how  to  improve  the  effectiveness  of  the  systems  engineers 


Permissions,  Protocols,  and  Data  Handling 

The  Helix  project  includes  face-to-face  interviews  as  a  primary  mode  of  data  collection.  Therefore,  necessary 
permissions  and  approvals  need  to  be  obtained  from  Institutional  Review  Board  (IRB)  and  relevant  authorities 
from  DoD.  Potential  interviewees  will  be  first  screened  for  his  or  her  eligibility  to  participate  in  the  Helix  project 
and  their  voluntary  consent  will  be  obtained  before  proceeding  with  the  interview.  In  addition,  organizations  will 
need  individual  Non-Disclosure  Agreements  (NDA)  to  be  signed  by  the  project  team  before  they  could  share  any 
institutional  data.  Organizations  may  also  have  their  policies  on  the  logistics  and  conduct  of  interviews,  since  all 
interviews  would  to  be  conducted  on-site. 

High  level  of  data  encryption  and  security  protocols  will  be  deployed  by  the  research  team  on  all  project  related 
data  that  will  be  stored,  shared  or  analyzed  by  the  team.  Additional  security  measures  will  be  taken  to  isolate 
personally  identifiable  information  (Pll)  that  will  be  stripped  off  all  data  before  any  analysis.  All  physical  items 
including  paper  documents  and  electronic  storage  devices  will  be  stored  securely. 

The  identity  of  individuals  or  their  organizations  will  never  be  reported.  All  analysis  results  will  only  be  presented 
in  an  anonymous  aggregated  manner. 
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Project  Plans  [May  2013  onwards] 


From  May  2013  onwards,  the  project  team  expects  to  begin  conducting  actual  data  collection  (from  face-to-face 
interviews  and  institutional  data).  Based  on  the  initial  data  collection,  the  research  methodology,  analysis  tools  & 
techniques,  and  working  procedures  for  data  handling  will  be  validated.  Initial  analysis  is  not  expected  to  provide 
much  insight  mainly  because  the  data  collected  will  be  limited.  But  as  data  continues  to  be  collected,  and  when 
there  is  sufficient  data  to  be  able  to  present  results  in  an  anonymous  aggregated  manner,  reports  will  be 
generated  by  the  project  team. 

Beginning  in  the  second  half  of  2013,  Helix  will  publish  quarterly  reports  addressing  the  main  research  questions, 
including  such  demographic  data  as  age,  education,  experiences,  roles,  and  activities.  Each  new  report  will  fold 
into  the  analysis  new  insights  gained  from  the  previous  quarter's  data  collection.  The  reports  will  identify  forces 
(such  as  experiences,  education,  and  competencies)  that  most  affect  workforce  effectiveness  and  what 
organizations  are  doing  to  address  those  forces.  Reports  will  include  anonymous  aggregated  data  only.  Based  on 
the  data,  Helix  will  be  able  to  benchmark  different  segments  of  the  community  and  even  individual  organizations. 

While  the  current  study  is  focused  on  the  systems  engineering  workforce  within  the  US  DoD  community, 
considerable  interest  has  been  shown  by  INCOSE  to  extend  this  study  to  non-DoD  and  non-US  companies 
(including  defense  communities  outside  the  US).  These  extensions  to  the  Helix  project  will  be  considered  in  2014. 
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Appendix  A:  Sample  Interview  Questions  (Individual  Level) 


Getting  to  know  you 

IND01  Do  you  consider  yourself  a  systems  engineer? 

IND02  What  are  you  responsible  for  in  your  current  position? 

IND03  What  activities  do  you  perform  most  frequently? 

IND04  What  are  the  most  important  activities  that  you  perform? 

IND05  Are  you  spending  the  right  amount  of  time  on  the  most  important  activities? 

IND06  What  were  your  expectations  when  you  first  took  this  position?  Does  that  align  with  what  you  do 
now? 

IND07  How  well  does  your  management  understand  and  appreciate  what  you  do? 

What  are  the  characteristics  of  systems  engineers? 

IND08  Establish  definition  of  systems  engineer  and  definition  of  systems  engineering  workforce  to  be  used 
in  the  rest  of  the  interview  (from  Helix,  organizational,  individual  definitions) 

IND09  Do  you  still  consider  yourself  a  systems  engineer  in  light  of  this  definition? 

INDIO  How  does  this  definition  of  a  systems  engineer  match  what  you  see  in  practice? 

IND11  What  are  typical  systems  engineering  activities  performed  in  your  organization?  How  are  they  related 
to  other  engineering  and  management  activities? 

IND12  How  common  is  it  for  people  to  perform  systems  engineering  activities  who  are  not  classified  as 
systems  engineers  and  vice-versa? 

How  effective  are  systems  engineers  and  why? 

IND13  Establish  definition  of  the  interviewee's  effectiveness  and  effectiveness  of  systems  engineering 

workforce  to  be  used  in  the  rest  of  the  interview  (from  Helix,  organizational,  individual  definitions) 
IND14  What  are  the  most  important  personal  traits  that  make  you  an  effective  systems  engineer?  Why? 
IND15  What  are  the  most  important  forces  that  increase  your  effectiveness  as  a  systems  engineer?  Why? 
IND16  What  are  the  most  important  forces  that  inhibit  your  effectiveness  as  a  systems  engineer?  Why? 
IND17  Do  you  think  that  the  impact  of  those  forces  is  uniform  across  the  systems  engineering  workforce? 
IND18  How  could  you  become  a  more  effective  systems  engineer? 

IND19  How  is  your  performance  as  a  systems  engineer  evaluated  and  rewarded?  What  metrics  are  used? 
IND20  Do  you  think  your  effectiveness  is  appropriately  considered  in  that  evaluation  and  reward? 

What  are  employers  doing  to  improve  their  effectiveness? 

IND21  What  personal  initiatives  have  you  been  taking  to  improve  your  own  effectiveness? 

IND22  Which  organizational  initiatives  in  the  last  5  years  have  been  helping  improve  your  effectiveness? 
IND23  Are  these  initiatives  adequate?  What  more  should  be  done? 
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Appendix  B:  Sample  Interview  Questions  (Organizational  Level) 


Getting  to  know  you 

ORGOl  Do  you  consider  yourself  a  systems  engineer? 

ORG02  What  are  you  responsible  for  in  your  current  position? 

ORG03  What  activities  do  you  perform  most  frequently? 

ORG04  What  are  the  most  important  activities  that  you  perform? 

ORG05  Are  you  spending  the  right  amount  of  time  on  the  most  important  activities? 

ORG06  What  were  your  expectations  when  you  first  took  this  position?  Does  that  align  with  what  you  do  now? 

ORG07  How  well  does  your  management  understand  and  appreciate  what  you  do? 

What  are  the  characteristics  of  systems  engineers? 

ORG08  Establish  definition  of  systems  engineer  and  definition  of  systems  engineering  workforce  to  be  used  in  the 

rest  of  the  interview  (from  Helix,  organizational,  individual  definitions) 

ORG09  Do  you  still  consider  yourself  a  systems  engineer  in  light  of  this  definition? 

ORGIO  How  does  this  definition  of  a  systems  engineer  match  what  you  see  in  practice? 

ORG11  What  are  typical  systems  engineering  activities  performed  in  your  organization?  How  are  they  related  to 
other  engineering  and  management  activities? 

ORG12  How  common  is  it  for  people  to  perform  systems  engineering  activities  who  are  not  classified  as  systems 
engineers  and  vice-versa? 

How  effective  are  systems  engineers  and  why? 

ORG13  Establish  definition  of  an  individual  systems  engineer's  effectiveness  and  effectiveness  of  systems 
engineering  workforce  to  be  used  in  the  rest  of  the  interview  (from  Helix,  organizational,  individual 
definitions) 

ORG14  What  are  the  most  important  personal  traits  that  make  a  systems  engineer  effective?  Why? 

ORG15  If  you  had  to  do  it  all  over  again,  what  would  you  do  differently? 

ORG16  What  are  the  most  important  forces  that  increase  the  effectiveness  of  a  systems  engineer?  Why? 

ORG17  What  are  the  most  important  forces  that  inhibit  the  effectiveness  of  a  systems  engineer?  Why? 

ORG18  Do  you  think  that  the  impact  of  those  forces  is  uniform  across  the  systems  engineering  workforce? 

ORG19  Is  there  a  gap  between  the  effectiveness  of  your  systems  engineering  workforce  and  your  organizational 
need? 

ORG20  How  is  the  performance  of  your  systems  engineering  workforce  evaluated  and  rewarded?  What  metrics 
are  used?  Are  they  uniform  across  the  roles  and  levels  of  the  systems  engineering  workforce? 

ORG21  Do  you  think  the  effectiveness  of  a  systems  engineer  is  appropriately  considered  in  that  evaluation  and 
reward? 

ORG22  What  are  the  primary  risks  to  the  systems  engineering  workforce  in  the  next  5  years?  How  are  these  risks 
being  addressed? 

What  are  employers  doing  to  improve  their  effectiveness? 

ORG23  What  personal  initiatives  have  you  been  taking  to  improve  your  own  effectiveness? 

ORG24  Which  organizational  initiatives  in  the  last  5  years  have  had  the  greatest  impact  on  the  forces  that 
improve  workforce  effectiveness?  How  do  you  know  this? 

ORG25  How  did  your  workforce  respond  to  these  initiatives? 

ORG26  Are  these  initiatives  adequate  to  close  the  gap  between  effectiveness  and  organizational  need?  What 
more  should  be  done? 
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Appendix  C:  Institutional  Data 


Institutional  data  requested  from  each  participating  organization: 

1.  The  charter  or  primary  purpose  of  the  organization 

2.  The  primary  business  of  the  organization,  including  revenue,  primary  customer,  organization-chart,  and 
types  of  products  and  services  delivered 

3.  Total  number  of  employees  in  the  organization  in  each  year  since  2009,  divided  into  engineers  and  non¬ 
engineers,  including  the  number  of  people  hired  and  departed 

4.  Total  number  of  systems  engineers  in  the  organization  in  each  year  since  2009  including  the  number  of 
people  hired  and  departed,  however  the  organization  defines  "systems  engineer" 

5.  A  characterization  of  the  systems  engineer  population  with  respect  to  highest  college  degree,  number  of 
years  of  professional  experience,  number  of  years  of  experience  as  a  systems  engineer,  age,  gender,  title 
or  rank  (such  as  systems  engineer,  senior  systems  engineer,  chief  systems  engineer,  etc.  using  whatever 
titles  or  ranks  that  are  part  of  the  human  resources  system),  and  years  to  retirement  eligibility 

6.  The  way  in  which  systems  engineers  are  primarily  organized;  e.g.,  in  a  matrix  structure  or  project 
structure 

7.  Major  organization  initiatives  now  underway  to  improve  the  quality  or  quantity  of  systems  engineers 

8.  Policies  that  are  particularly  relevant  to  systems  engineers,  including  organizational  competency  model, 
definition  of  systems  engineer,  and  career  paths 

9.  Major  SE-relevant  workforce  initiatives  and  how  the  organization  is  tracking  their  impact. 

10.  Overview  of  how  evaluation  is  performed  within  the  organization  (process,  metrics,  etc.) 
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